convert column table to arrow if arrow present #551
Conversation
|
Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase ( |
There was a problem hiding this comment.
Pull Request Overview
This PR updates the behavior of fetching query results by converting a ColumnTable to a PyArrow Table when PyArrow is available. It adds a new end-to-end test to validate that the catalogs() API returns a PyArrow Table and updates the fetchall_arrow() method in the client to merge columnar results when appropriate.
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| tests/e2e/test_driver.py | Added test_catalogs_returns_arrow_table to verify arrow table conversion. |
| src/databricks/sql/client.py | Updated fetchall_arrow() to merge ColumnTables and convert the final result where PyArrow is present. |
shivam2680
requested review from
deeksha-db,
samikshya-db,
yunbodeng-db,
jackyhu-db and
benc-db
as code owners
|
can you add more description in the PR to explain why you make this change? Is there any disruption risk of this change? |
Updated description |
| self._next_row_index += partial_results.num_rows | ||
|
|
||
| # If PyArrow is installed and we have a ColumnTable result, | ||
| # convert it to a PyArrow Table for consistency with pre-3.5.0 behavior | ||
| if isinstance(results, ColumnTable) and pyarrow: |
There was a problem hiding this comment.
this seems a breaking change? Does it impact the client uses 3.5.0 or higher
What type of PR is this?
Description
Added logic in
fetchall_arrow()to convertColumnTableresults to apyarrow.Tableifpyarrowis installed, maintaining consistency with behavior prior to version 3.5.0. This behavior makes result set aligned with the docstring offetchall_pyarrowmethodHow is this tested?
Introduced a new test case,
test_catalogs_returns_arrow_table, intests/e2e/test_driver.pyto validate that thefetchall_arrowmethod returns apyarrow.Tablewhen used withcursor.catalogs(). This test is conditionally executed if Arrow support is available.Related Tickets & Documents
Resolves #550